{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X6-q-gTUaZU7",
    "tags": []
   },
   "source": [
    "# Lesson 2: RAG Triad of metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "height": 98
   },
   "outputs": [],
   "source": [
    "import utils\n",
    "\n",
    "import os\n",
    "import openai\n",
    "openai.api_key = utils.get_openai_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "height": 81,
    "id": "IBfdyn3MaZU9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
      "🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.\n"
     ]
    }
   ],
   "source": [
    "from trulens_eval import Tru\n",
    "\n",
    "tru = Tru()\n",
    "tru.reset_database()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "height": 98,
    "id": "wMvq1q8yaZU-"
   },
   "outputs": [],
   "source": [
    "from llama_index import SimpleDirectoryReader\n",
    "\n",
    "documents = SimpleDirectoryReader(\n",
    "    input_files=[\"./eBook-How-to-Build-a-Career-in-AI.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41,\n",
       " Document(id_='5c8a5065-31ab-4c81-ae43-31d3a319db7b', embedding=None, metadata={'page_label': '1', 'file_name': 'eBook-How-to-Build-a-Career-in-AI.pdf', 'file_path': 'eBook-How-to-Build-a-Career-in-AI.pdf', 'file_type': 'application/pdf', 'file_size': 3717673, 'creation_date': '2024-11-23', 'last_modified_date': '2024-07-29', 'last_accessed_date': '2024-07-29'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={}, hash='5b458e7cf050778d0c8199cb37302c1129e02166a13bed6cddaa132be5c8c290', text='PAGE 1Founder, DeepLearning.AICollected Insights\\nfrom Andrew Ng\\nHow to \\nBuild\\nYour\\nCareer\\nin AIA Simple Guide\\n', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(documents), documents[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "height": 81,
    "id": "sY8Oui4taZU-"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(id_='3750e7b2-dd97-4d8f-a87e-bbd70b908260', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='fdc7ce9a017ce62d4e0b30cc419051a9bb56618a8569f51a2b64f8535fe6244e', text='PAGE 1Founder, DeepLearning.AICollected Insights\\nfrom Andrew Ng\\nHow to \\nBuild\\nYour\\nCareer\\nin AIA Simple Guide\\n\\n\\nPAGE 2\"AI is the new \\nelectricity. It will \\ntransform and improve \\nall areas of human life.\"\\nAndrew Ng\\n\\nPAGE 3Table of \\nContentsIntroduction: Coding AI is the New Literacy.\\nChapter 1: Three Steps to Career Growth.\\nChapter 2: Learning Technical Skills for a \\nPromising AI Career.\\nChapter 3: Should You Learn Math to Get a Job \\nin AI?\\nChapter 4: Scoping Successful AI Projects.\\nChapter 5: Finding Projects that Complement \\nYour Career Goals.\\nChapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\nChapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\nChapter 8: Using Informational Interviews to Find \\nthe Right Job.\\nChapter 9: Finding the Right AI Job for You.\\nChapter 10: Keys to Building a Career in AI.\\nChapter 11: Overcoming Imposter Syndrome.\\nFinal Thoughts: Make Every Day Count.LEARNING\\nPROJECTS\\nJOB\\n\\nPAGE 4Coding AI Is the New Literacy\\nToday we take it for granted that many people know how to read and write. Someday, I hope, \\nit will be just as common that people know how to write code, specifically for AI.\\nSeveral hundred years ago, society didn’t view language literacy as a necessary skill. A small \\nnumber of people learned to read and write, and everyone else let them do the reading and \\nwriting. It took centuries for literacy to spread, and now society is far richer for it.\\nWords enable deep human-to-human communication. Code is the deepest form of human-to-\\nmachine communication. As machines become more central to daily life, that communication \\nbecomes ever more important.\\nTraditional software engineering — writing programs that explicitly tell a computer sequences \\nof steps to execute — has been the main path to code literacy. Many introductory programming \\nclasses use creating a video game or building a website as examples. But AI, machine learning, \\nand data science offer a new paradigm in which computers extract knowledge from data. This \\ntechnology offers an even better pathway to coding.\\nMany Sundays, I buy a slice of pizza from my neighborhood pizza parlor. The gentleman \\nbehind the counter has little reason to learn how to build a video game or write his own \\nwebsite software (beyond personal growth and the pleasure of gaining a new skill).\\nBut AI and data science have great value even for a pizza maker. A linear regression model might \\nenable him to better estimate demand so he can optimize the restaurant’s staffing and supply \\nchain. He could better predict sales of Hawaiian pizza — my favorite! — so he could make more \\nHawaiian pies in advance and reduce the amount of time customers had to wait for them.\\nUses of AI and data science can be found in almost any situation that produces data.  Thus, \\na wide variety of professions will find more uses for custom AI applications and data-derived \\ninsights than for traditional software engineering. This makes literacy in AI-oriented coding \\neven more valuable than traditional coding. It could enable countless individuals to harness \\ndata to make their lives richer.\\nI hope the promise of building basic AI applications, even more than that of building basic \\ntraditional software, encourages more people to learn how to code. If society embraces this \\nnew form of literacy as it has the ability to read and write, we will all benefit.Introduction\\n\\nPAGE 5Three Steps to \\nCareer GrowthCHAPTER 1\\n\\nPAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building exciting \\ncareers in this field. A career is a decades-long journey, and the path is not straightforward. \\nOver many years, I’ve been privileged to see thousands of students, as well as engineers in \\ncompanies large and small, navigate careers in AI.\\n \\nHere’s a framework for charting your own course.\\nLater, you will work \\non finding a job. \\nThroughout this \\nprocess, you’ll continue \\nto learn and work on \\nmeaningful projects. \\nChapters with the\\nfocus on a job search.Three key steps of career growth are learning foundational skills, working on projects (to \\ndeepen your skills, build a portfolio, and create impact), and finding a job. These steps stack \\non top of each other:Three Steps to Career Growth\\nLEARNING PROJECTS JOBCHAPTER 1\\nInitially, you \\nfocus on learning \\nfoundational skills. \\nChapters with the \\ncover topics about \\nlearning foundational \\ntechnical skills. After having gained \\nfoundational technical \\nskills, you will begin \\nworking on projects. \\nDuring this period, you’ll \\nalso keep learning. \\nChapters with the     \\nfocus on projects. \\n\\nPAGE 7These phases apply in a wide \\nrange of professions, but AI \\ninvolves unique elements.\\nFor example:Three Steps to Career Growth\\nAI is nascent, and many technologies are still evolving. While the \\nfoundations of machine learning and deep learning are maturing — \\nand coursework is an efficient way to master them — beyond these \\nfoundations, keeping up-to-date with changing technology is more \\nimportant in AI than fields that are more mature.Learning foundational skills is a career-long process:\\nThis can make it challenging to find a suitable project, estimate the project’s \\ntimeline and return on investment, and set expectations. In addition, the \\nhighly iterative nature of AI projects leads to special challenges in project \\nmanagement: How can you come up with a plan for building a system \\nwhen you don’t know in advance how long it will take to achieve the target \\naccuracy? Even after the system has hit the target, further iteration may \\nbe necessary to address post-deployment drift.Working on projects often means collaborating with \\nstakeholders who lack expertise in AI:\\nWhile searching for a job in AI can be similar to searching for a job in \\nother sectors, there are also important differences. Many companies are \\nstill trying to figure out which AI skills they need, and how to hire people \\nwho have them. Things you’ve worked on may be significantly different \\nthan anything your interviewer has seen, and you’re more likely to have to \\neducate potential employers about some elements of your work.Inconsistent opinions on AI skills and jobs roles: CHAPTER 1\\nAs you go through each step, you should also build a supportive community. Having friends and \\nallies who can help you — and who you strive to help — makes the path easier. This is true whether \\nyou’re taking your first steps or you’ve been on the journey for years.LEARNING\\nPROJECTS\\nJOB\\n\\nPAGE 8Learning Technical \\nSkills for a Promising \\nAI CareerCHAPTER 2\\nLEARNING\\n\\nPAGE 9In the previous chapter, I introduced three key steps for building a career in AI: learning \\nfoundational technical skills, working on projects, and finding a job, all of which is supported \\nby being part of a community. In this chapter, I’d like to dive more deeply into the first step: \\nlearning foundational skills.\\nMore research papers have been published on AI than anyone can read in a lifetime. So, when \\nlearning, it’s critical to prioritize topic selection. I believe the most important topics for a technical \\ncareer in machine learning are:\\nFoundational machine learning skills: For example, it’s important to understand models such \\nas linear regression, logistic regression, neural networks, decision trees, clustering, and anomaly \\ndetection. Beyond specific models, it’s even more important to understand the core concepts \\nbehind how and why machine learning works, such as bias/variance, cost functions, regularization, \\noptimization algorithms, and error analysis.\\nDeep learning: This has become such a large fraction of machine learning that it’s hard to excel \\nin the field without some understanding of it! It’s valuable to know the basics of neural networks, \\npractical skills for making them work (such as hyperparameter tuning), convolutional networks, \\nsequence models, and transformers.\\nSoftware development: While you can get a job and make huge contributions with only machine \\nlearning modeling skills, your job opportunities will increase if you can also write good software \\nto implement complex AI systems. These skills include programming fundamentals, data \\nstructures (especially those that relate to machine learning, such as data frames), algorithms \\n(including those related to databases and data manipulation), software design, familiarity with \\nPython, and familiarity with key libraries such as TensorFlow or PyTorch, and scikit-learn.Math relevant to machine learning:  Key areas include linear algebra (vectors, matrices, and various \\nmanipulations of them) as well as probability and statistics (including discrete and continuous \\nprobability, standard probability distributions, basic rules such as independence and Bayes’ rule, \\nand hypothesis testing). In addition, exploratory data analysis (EDA) — using visualizations and other \\nmethods to systematically explore a dataset — is an underrated skill. I’ve found EDA particularly \\nuseful in data-centric AI development, where analyzing errors and gaining insights can really help \\ndrive progress! Finally, a basic intuitive understanding of calculus will also help. The math needed \\nto do machine learning well has been changing. For instance, although some tasks require calculus, \\nimproved automatic differentiation software makes it possible to invent and implement new neural \\nnetwork architectures without doing any calculus. This was almost impossible a decade ago.Learning Technical Skills For a Promising AI Career CHAPTER 2\\n\\nPAGE 10This is a lot to learn!\\nEven after you master everything on this list, I hope you’ll keep learning and continue to deepen \\nyour technical knowledge. I’ve known many machine learning engineers who benefitted from \\ndeeper skills in an application area such as natural language processing or computer vision, or in \\na technology area such as probabilistic graphical models or building scalable software systems.\\nHow do you gain these skills? There’s a lot of good content on the internet, and in theory, \\nreading dozens of web pages could work. But when the goal is deep understanding, reading \\ndisjointed web pages is inefficient because they tend to repeat each other, use inconsistent \\nterminology (which slows you down), vary in quality, and leave gaps. That’s why a good course \\n— in which a body of material has been organized into a coherent and logical form — is often the \\nmost time-efficient way to master a meaningful body of knowledge. When you’ve absorbed the \\nknowledge available in courses, you can switch over to research papers and other resources.\\nFinally, no one can cram everything they need to know over a weekend or even a month. Everyone I \\nknow who’s great at machine learning is a lifelong learner. Given how quickly our field is changing, \\nthere’s little choice but to keep learning if you want to keep up.\\nHow can you maintain a steady pace of learning for years? If you can cultivate the habit of \\nlearning a little bit every week, you can make significant progress with what feels like less effort.Learning Technical Skills For a Promising AI Career CHAPTER 2\\n\\n\\nPAGE 11\\nThe Best Way to Build \\na New Habit\\nOne of my favorite books is BJ Fogg’s, Tiny Habits: The Small Changes That Change \\nEverything. Fogg explains that the best way to build a new habit is to start small \\nand succeed, rather than start  too big and fail. For example, rather than trying to \\nexercise for 30 minutes a day, he recommends aspiring to do just one push-up, and \\ndoing it consistently.\\nThis approach may be helpful to those of you who want to spend more time studying. \\nIf you start by holding yourself accountable for watching, say, 10 seconds of an \\neducational video every day — and you do so consistently — the habit of studying daily \\nwill grow naturally. Even if you learn nothing in that 10 seconds, you’re establishing the \\nhabit of studying a little every day. On some days, maybe you’ll end up studying for an \\nhour or longer.\\n\\nPAGE 12Should You \\nLearn Math to \\nGet a Job in AI? CHAPTER 3\\nLEARNING\\n\\nPAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\\nIs math a foundational skill for AI? It’s always nice to know more math! But there’s so much to \\nlearn that, realistically, it’s necessary to prioritize. Here’s how you might go about strengthening \\nyour math background.\\nTo figure out what’s important to know, I find it useful to ask what you need to know to make \\nthe decisions required for the work you want to do. At DeepLearning.AI, we frequently ask, \\n“What does someone need to know to accomplish their goals?” The goal might be building a \\nmachine learning model, architecting a system, or passing a job interview.\\nUnderstanding the math behind algorithms you use is often helpful, since it enables you to \\ndebug them. But the depth of knowledge that’s useful changes over time. As machine learning \\ntechniques mature and become more reliable and turnkey, they require less debugging, and a \\nshallower understanding of the math involved may be sufficient to make them work.\\nFor instance, in an earlier era of machine learning, linear algebra libraries for solving linear \\nsystems of equations (for linear regression) were immature. I had to understand how these \\nlibraries worked so I could choose among different libraries and avoid numerical roundoff \\npitfalls. But this became less important as numerical linear algebra libraries matured.\\nDeep learning is still an emerging technology, so when you train a neural network and the \\noptimization algorithm struggles to converge, understanding the math behind gradient \\ndescent, momentum, and the Adam  optimization algorithm will help you make better decisions. \\nSimilarly, if your neural network does something funny — say, it makes bad predictions on \\nimages of a certain resolution, but not others — understanding the math behind neural network \\narchitectures puts you in a better position to figure out what to do.\\nOf course, I also encourage learning driven by curiosity. If something interests you, go ahead \\nand learn it regardless of how useful it might turn out to be!  Maybe this will lead to a creative \\nspark or technical breakthrough.How much math do you need to know to be a machine learning engineer?\\n\\nPAGE 14Scoping Successful \\nAI ProjectsCHAPTER 4\\nPROJECTS\\n\\nPAGE 15One of the most important skills of an AI architect is the ability to identify ideas that are worth \\nworking on. These next few chapters will discuss finding and working on projects so you can gain \\nexperience and build your portfolio. \\nOver the years, I’ve had fun applying machine learning to manufacturing, healthcare, climate \\nchange, agriculture, ecommerce, advertising, and other industries. How can someone who’s not \\nan expert in all these sectors find meaningful projects within them? Here are five steps to help \\nyou scope projects.\\nIdentify a business problem (not an AI problem). I like to find \\na domain expert and ask, “What are the top three things \\nthat you wish worked better? Why aren’t they working yet?” \\nFor example, if you want to apply AI to climate change, you \\nmight discover that power-grid operators can’t accurately \\npredict how much power intermittent sources like wind \\nand solar might generate in the future.\\nBrainstorm AI solutions. When I was younger, I used to \\nexecute on the first idea I was excited about. Sometimes \\nthis worked out okay, but sometimes I ended up missing \\nan even better idea that might not have taken any more \\neffort to build. Once you understand a problem, you can \\nbrainstorm potential solutions more efficiently. For instance, \\nto predict power generation from intermittent sources, we \\nmight consider using satellite imagery to map the locations \\nof wind turbines more accurately, using satellite imagery \\nto estimate the height and generation capacity of wind \\nturbines, or using weather data to better predict cloud cover \\nand thus solar irradiance. Sometimes there isn’t a good AI \\nsolution, and that’s okay too.Scoping Successful AI Projects CHAPTER 4\\nStep 1\\nStep 2\\n\\n\\nPAGE 16Determine milestones. Once you’ve deemed a project sufficiently \\nvaluable, the next step is to determine the metrics to aim for. This \\nincludes both machine learning metrics (such as accuracy) and \\nbusiness metrics (such as revenue). Machine learning teams are often \\nmost comfortable with metrics that a learning algorithm can optimize. \\nBut we may need to stretch outside our comfort zone to come up \\nwith business metrics, such as those related to user engagement, \\nrevenue, and so on. Unfortunately, not every business problem can be \\nreduced to optimizing test set accuracy! If you aren’t able to determine \\nreasonable milestones, it may be a sign that you need to learn more \\nabout the problem. A quick proof of concept can help supply the \\nmissing perspective.Assess the feasibility and value of potential solutions. You can determine \\nwhether an approach is technically feasible by looking at published work, \\nwhat competitors have done, or perhaps building a quick proof of concept \\nimplementation. You can determine its value by consulting with domain \\nexperts (say, power-grid operators, who can advise on the utility of the \\npotential solutions mentioned above).\\nBudget for resources. Think through everything you’ll need to get the \\nproject done including data, personnel, time, and any integrations or \\nsupport you may need from other teams. For example, if you need funds \\nto purchase satellite imagery, make sure that’s in the budget.\\nWorking on projects is an iterative process. If, at any step, you find that the current direction is \\ninfeasible, return to an earlier step and proceed with your new understanding. Is there a domain \\nthat excites you where AI might make a difference? I hope these steps will guide you in exploring it \\nthrough project work — even if you don’t yet have deep expertise in that field. AI won’t solve every \\nproblem, but as a community, let’s look for ways to make a positive impact wherever we can.Scoping Successful AI Projects CHAPTER 4\\nStep 3\\nStep 4\\nStep 5\\n\\nPAGE 17Finding Projects that \\nComplement Your \\nCareer GoalsCHAPTER 5\\nPROJECTS\\n\\nPAGE 18It goes without saying that we should only work on projects that are responsible, ethical, and \\nbeneficial to people. But those limits leave a large variety to choose from. In the previous chapter, \\nI wrote about how to identify and scope AI projects. This chapter and the next have a slightly \\ndifferent emphasis: picking and executing projects with an eye toward career development.\\nA fruitful career will include many projects, hopefully growing in scope, complexity, and impact \\nover time. Thus, it is fine to start small. Use early projects to learn and gradually step up to \\nbigger projects as your skills grow.\\nWhen you’re starting out, don’t expect others to hand great ideas or resources to you on a platter. \\nMany people start by working on small projects in their spare time. With initial successes — even \\nsmall ones — under your belt, your growing skills increase your ability to come up with better \\nideas, and it becomes easier to persuade others to help you step up to bigger projects.Finding Projects that Compliment Your Career Goals CHAPTER 5\\nJoin existing projects. If you find someone else with an idea, ask to join their project.\\nKeep reading and talking to people. I come up with new ideas whenever I spend a lot of \\ntime reading, taking courses, or talking with domain experts. I’m confident that you will, too.\\nFocus on an application area. Many researchers are trying to advance basic AI technology \\n— say, by inventing the next generation of transformers or further scaling up language \\nmodels — so, while this is an exciting direction, it is also very hard. But the variety of \\napplications to which machine learning has not yet been applied is vast! I’m fortunate to \\nhave been able to apply neural networks to everything from autonomous helicopter flight to \\nonline advertising, partly because I jumped in when relatively few people were working on \\nthose applications. If your company or school cares about a particular application, explore \\nthe possibilities for machine learning. That can give you a first look at a potentially creative \\napplication — one where you can do unique work — that no one else has done yet.✓\\n✓\\n✓What if you don’t have any project ideas?\\nHere are a few ways to generate them:\\n\\nPAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not \\ndevelop into something bigger can stir the creative juices and strengthen bonds with \\ncollaborators. When I was a full-time professor, working on online education wasn’t part of \\nmy “job” (which was doing research and teaching classes). It was a fun hobby that I often \\nworked on out of passion for education. My early experiences in recording videos at home \\nhelped me later in working on online education in a more substantive way. Silicon Valley \\nabounds with stories of startups that started as side projects. As long as it doesn’t create a \\nconflict with your employer, these projects can be a stepping stone to something significant.\\nWill the project help you grow technically? Ideally, it should be challenging enough to \\nstretch your skills but not so hard that you have little chance of success. This will put you \\non a path toward mastering ever-greater technical complexity.\\nDo you have good teammates to work with? If not, are there people you can discuss things \\nwith? We learn a lot from the people around us, and good collaborators will have a huge \\nimpact on your growth.\\nCan it be a stepping stone?  If the project is successful, will its technical complexity and/\\nor business impact make it a meaningful stepping stone to larger projects? If the project \\nis bigger than those you’ve worked on before, there’s a good chance it could be such a \\nstepping stone.✓\\n✓\\n✓\\n✓Given a few project ideas, which one should you jump into? \\nHere’s a quick checklist of factors to consider:\\nFinally, avoid analysis paralysis. It doesn’t make sense to spend a month deciding whether to \\nwork on a project that would take a week to complete. You\\'ll work on multiple projects over \\nthe course of your career, so you’ll have ample opportunity to refine your thinking on what’s \\nworthwhile. Given the huge number of possible AI projects, rather than the conventional “ready, \\naim, fire” approach, you can accelerate your progress with “ready, fire, aim.”Finding Projects that Compliment Your Career Goals CHAPTER 5\\n\\nPAGE 20Working on projects requires making tough choices about what to build and how to go \\nabout it. Here are two distinct styles:\\nSay you’ve built a customer-service chatbot for retailers, and you think it could help restaurants, \\ntoo. Should you take time to study the restaurant market before starting development, moving \\nslowly but cutting the risk of wasting time and resources? Or jump in right away, moving \\nquickly and accepting a higher risk of pivoting or failing?\\nBoth approaches have their advocates, and the best choice depends on the situation.\\nReady, Aim, Fire tends to be superior when the cost of execution is high and a study can shed \\nlight on how useful or valuable a project could be. For example, if you can brainstorm a few \\nother use cases (restaurants, airlines, telcos, and so on) and evaluate these cases to identify \\nthe most promising one, it may be worth taking the extra time before committing to a direction.\\nReady, Fire, Aim tends to be better if you can execute at low cost and, in doing so, determine \\nwhether the direction is feasible and discover tweaks that will make it work. For example, if \\nyou can build a prototype quickly to figure out if users want the product, and if canceling or \\npivoting after a small amount of work is acceptable, then it makes sense to consider jumping \\nin quickly. When taking a shot is inexpensive, it also makes sense to take many shots. In this \\ncase, the process is actually Ready, Fire, Aim, Fire, Aim, Fire, Aim, Fire. \\nAfter agreeing upon a project direction, when it comes to building a machine learning model \\nthat’s part of the product, I have a bias toward Ready, Fire, Aim. Building models is an iterative \\nprocess. For many applications, the cost of training and conducting error analysis is not \\nprohibitive. Furthermore, it is very difficult to carry out a study that will shed light on the \\nappropriate model, data, and hyperparameters. So it makes sense to build an end-to-end \\nsystem quickly and revise it until it works well.\\nBut when committing to a direction means making a costly investment or entering a one-\\nway door  (meaning a decision that’s hard to reverse), it’s often worth spending more time in \\nadvance to make sure it really is a good idea.Ready, Fire, Aim\\nReady, Aim, Fire: Plan carefully and carry out careful validation. Commit and \\nexecute only when you have a high degree of confidence in a direction.\\nReady, Fire, Aim: Jump into development and start executing. This allows you to \\ndiscover problems quickly and pivot along the way if necessary.✓\\n✓\\n\\nPAGE 21Building a Portfolio of \\nProjects that Shows \\nSkill Progression CHAPTER 6\\nPROJECTS\\n\\nPAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing in \\nscope and complexity. For example:\\nThe first few projects might be narrowly scoped \\nhomework assignments with predetermined right \\nanswers. These are often great learning experiences!\\nEventually, you will gain enough skill to build projects in \\nwhich others see more tangible value. This opens the door \\nto more resources. For example, rather than developing \\nmachine learning systems in your spare time, it might \\nbecome part of your job, and you might gain access to more \\nequipment, compute time, labeling budget, or head count.You might go on to work on small-scale projects either alone or with friends. \\nFor instance, you might re-implement a known algorithm, apply machine \\nlearning to a hobby (such as predicting whether your favorite sports team \\nwill win), or build a small but useful system at work in your spare time (such \\nas a machine learning-based script that helps a colleague automate some of \\ntheir work). Participating in competitions such as those organized by Kaggle \\nis also one way to gain experience.\\nSuccesses build on each other, opening the door to \\nmore technical growth, more resources, and increasingly \\nsignificant project opportunities.1. Class projects:\\n3. Creating value2. Personal projects\\n4. Rising scope and complexityBuilding a Portfolio of Projects That Shows Skill Progression CHAPTER 6\\n\\nPAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impact. In addition:\\nDon’t worry about starting too small. One of my first machine learning research projects involved \\ntraining a neural network to see how well it could mimic the sin(x) function. It wasn’t very useful, but \\nwas a great learning experience that enabled me to move on to bigger projects.\\nBuilding a portfolio of projects, especially one \\nthat shows progress over time from simple to \\ncomplex undertakings, will be a big help when \\nit comes to looking for a job.Communication is key.  You need to be able to explain your thinking if you want others to see \\nthe value in your work and trust you with resources that you can invest in larger projects. To get \\na project started, communicating the value of what you hope to build will help bring colleagues, \\nmentors, and managers onboard — and help them point out flaws in your reasoning. After you’ve \\nfinished, the ability to explain clearly what you accomplished will help convince others to open the \\ndoor to larger projects.\\nLeadership isn’t just for managers.  When you reach the point of working on larger AI projects that \\nrequire teamwork, your ability to lead projects will become more important, whether or not you are \\nin a formal position of leadership. Many of my friends have successfully pursued a technical rather \\nthan managerial career, and their ability to help steer a project by applying deep technical insights \\n— for example, when to invest in a new technical architecture or collect more data of a certain type \\n— allowed them to grow as leaders and also helped significantly improve the project.Building a Portfolio of Projects That Shows Skill Progression CHAPTER 6\\n\\nPAGE 24A Simple Framework \\nfor Starting Your AI \\nJob SearchCHAPTER 7\\nJOBS\\n\\nPAGE 25Finding a job has a few predictable steps that include selecting the companies to which you \\nwant to apply, preparing for interviews, and finally picking a role and negotiating a salary and \\nbenefits. In this chapter, I’d like to focus on a framework that’s useful for many job seekers in \\nAI, especially those who are entering AI from a different field.\\nA product manager at a tech startup who becomes a data scientist at the same company (or a \\ndifferent one) has switched roles. A marketer at a manufacturing firm who becomes a marketer \\nin a tech company has switched industries. An analyst in a financial services company who \\nbecomes a machine learning engineer in a tech company has switched both roles and industries.\\nIf you’re looking for your first job in AI, you’ll probably find switching either roles or industries \\neasier than doing both at the same time. Let’s say you’re the analyst working in financial services:If you’re considering your next job, ask yourself:\\nJob \\nSearchAre you switching roles? For example, if you’re a software engineer, \\nuniversity student, or physicist who’s looking to become a machine learning \\nengineer, that’s a role switch.\\nAre you switching industries? For example, if you work for a healthcare \\ncompany, financial services company, or a government agency and want \\nto work for a software company, that’s a switch in industries.✓\\n✓\\nIf you find a data science or machine learning job in financial services, you can continue \\nto use your domain-specific knowledge while gaining knowledge and expertise in AI. After \\nworking in this role for a while, you’ll be better positioned to switch to a tech company (if \\nthat’s still your goal).\\nAlternatively, if you become an analyst in a tech company, you can continue to use your \\nskills as an analyst but apply them to a different industry. Being part of a tech company also \\nmakes it much easier to learn from colleagues about practical challenges of AI, key skills to \\nbe successful in AI, and so on.✓\\n✓A Simple Framework for Starting You AI Job Search CHAPTER 7\\nTECHFINANCIAL \\nSERVICESRole & Industry SwitchRole Switch\\nIndustry SwitchANALYSTMACHINE LEARNING \\nENGINEER\\n\\nPAGE 26If you’re considering a role switch, a startup can be an easier place to do it than a big company. \\nWhile there are exceptions, startups usually don’t have enough people to do all the desired \\nwork. If you’re able to help with AI tasks — even if it’s not your official job — your work is likely \\nto be appreciated. This lays the groundwork for a possible role switch without needing to leave \\nthe company. In contrast, in a big company, a rigid reward system is more likely to reward you \\nfor doing your job well (and your manager for supporting you in doing the job for which you were \\nhired), but it’s not as likely to reward contributions outside your job’s scope.\\nAfter working for a while in your desired role and industry (for example, a machine learning \\nengineer in a tech company), you’ll have a good sense of the requirements for that role in that \\nindustry at a more senior level. You’ll also have a network within that industry to help you along. \\nSo future job searches — if you choose to stick with the role and industry — likely will be easier.\\nWhen changing jobs, you’re taking a step into the unknown, particularly if you’re switching either \\nroles or industries. One of the most underused tools for becoming more familiar with a new role \\nand/or industry is the informational interview. I’ll share more about that in the next chapter.\\nI’m grateful to Salwa Nur Muhammad, CEO of FourthBrain (a DeepLearning.AI affiliate), for providing \\nsome of the ideas presented in this chapter.\\n\\n\\nPAGE 27There’s a lot we don’t know about the future: When will we cure Alzheimer’s disease? Who will \\nwin the next election? Or, in a business context, how many customers will we have next year?\\nWith so many changes going on in the world, many people are feeling stressed about the \\nfuture, especially when it comes to finding a job. I have a practice that helps me regain a sense \\nof control. Faced with uncertainty, I try to:\\nFor example, during the Covid-19 pandemic back in March 2020, I did this scenario planning \\nexercise. I imagined quick (three months), medium (one year), and slow (two years) recoveries \\nfrom Covid-19 and made plans for managing each case. These plans have helped me prioritize \\nwhere I can.\\nThe same method can apply to personal life, too. If you’re not sure you’ll pass an exam, get a \\njob offer, or be granted a visa — all of which can be stressful — you can write out what you’d \\ndo in each of the likely scenarios. Thinking through the possibilities and following through on \\nplans can help you navigate the future effectively no matter what it brings.\\nBonus: With training in AI and statistics, you can calculate a probability for each scenario. \\nI’m a fan of the Superforecasting methodology, in which the judgments of many experts are \\nsynthesized into a probability estimate. Overcoming Uncertainty\\nMake a list of plausible scenarios, \\nacknowledging that I don’t know \\nwhich will come to pass.Create a plan of action \\nfor each scenario.\\nStart executing actions \\nthat seem reasonable.Review scenarios and plans \\nperiodically as the future \\ncomes into focus.1 2\\n3 4\\n\\nPAGE 28Using Informational \\nInterviews to Find \\nthe Right JobCHAPTER 8\\nJOBS\\n\\nPAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for the first \\ntime) or industries (say, working in an AI tech company for the first time), there’s a lot about \\nyour target job that you probably don’t know. A technique known as informational interviewing \\nis a great way to learn.\\nAn informational interview involves finding someone in a company or role you’d like to know \\nmore about and informally interviewing them about their work. Such conversations are separate \\nfrom searching for a job. In fact, it’s helpful to interview people who hold positions that align \\nwith your interests well before you’re ready to kick off a job search.\\nPrepare for informational interviews by researching the interviewee and company in advance, \\nso you can arrive with thoughtful questions. You might ask:Informational interviews are particularly relevant to AI. Because the field is evolving, many \\ncompanies use job titles in inconsistent ways. In one company, data scientists might be \\nexpected mainly to analyze business data and present conclusions on a slide deck. In \\nanother, they might write and maintain production code. An informational interview can help \\nyou sort out what the AI people in a particular company actually do.\\nWith the rapid expansion of opportunities in AI, many people will be taking on an AI job for \\nthe first time. In this case, an informational interview can be invaluable for learning what \\nhappens and what skills are needed to do the job well. For example, you can learn what \\nalgorithms, deployment processes, and software stacks a particular company uses. You \\nmay be surprised — if you’re not already familiar with the data-centric AI movement — to \\nlearn how much time most machine learning engineers spend iteratively cleaning datasets.✓\\n✓\\nWhat do you do in a typical week or day?\\nWhat are the most important tasks in this role?\\nWhat skills are most important for success?\\nHow does your team work together to accomplish its goals?\\nWhat is the hiring process?\\nConsidering candidates who stood out in the past, what enabled them to shine?✓\\n✓\\n✓\\n✓\\n✓\\n✓Using Informational Interviews to Find the Right Job CHAPTER 8\\n\\nPAGE 30Finding someone to interview isn’t always easy, but many people who are in senior positions today \\nreceived help when they were new from those who had entered the field ahead of them, and many \\nare eager to pay it forward. If you can reach out to someone who’s already in your network — \\nperhaps a friend who made the transition ahead of you or someone who attended the same school \\nas you — that’s great! Meetups such as Pie & AI can also help you build your network.\\nFinally, be polite and professional, and thank the people you’ve interviewed. And when you get \\na chance, please pay it forward as well and help someone coming up after you. If you receive \\na request for an informational interview from someone in the DeepLearning.AI community, \\nI hope you’ll lean in to help them take a step up! If you’re interested in learning more about \\ninformational interviews, I recommend this article from the UC Berkeley Career Center.\\nI’ve mentioned a few times the importance of your network and community. People you’ve \\nmet, beyond providing valuable information, can also play an invaluable role by referring you to \\npotential employers. Using Informational Interviews to Find the Right Job CHAPTER 8\\n\\n\\nPAGE 31Finding the Right \\nAI Job for YouCHAPTER 9\\nJOBS\\n\\nPAGE 32In this chapter, I’d like to \\ndiscuss some fine points \\nof finding a job.\\nThe typical job search follows a fairly predictable path.\\nAlthough the process may be familiar, every job search is different. Here are some tips to increase \\nthe odds you’ll find a position that supports your thriving career and enables you to keep growing.Research roles and companies online or by talking to friends.\\nOptionally, arrange informal informational interviews with people in companies that appeal to you.\\nEither apply directly or, if you can, get a referral from someone on the inside.\\nInterview with companies that give you an invitation.\\nReceive one or more offers and pick one. Or, if you don’t receive an offer, ask for feedback \\nfrom the interviewers, human resources staff, online discussion boards, or anyone in your \\nnetwork who can help you plot your next move.✓\\n✓\\n✓\\n✓\\n✓\\nPay attention to the fundamentals.  A compelling resume, portfolio of technical projects, and \\na strong interview performance will unlock doors. Even if you have a referral from someone in a \\ncompany, a resume and portfolio will be your first contact with many people who don’t already \\nknow about you. Update your resume and make sure it clearly presents your education and \\nexperience relevant to the role you want. Customize your communications with each company \\nto explain why you’re a good fit. Before an interview, ask the recruiter what to expect. Take time \\nto review and practice answers to common interview questions, brush up key skills, and study \\ntechnical materials to make sure they are fresh in your mind. Afterward, take notes to help you \\nremember what was said.\\nProceed respectfully and responsibly. Approach interviews and offer negotiations with a win-\\nwin mindset. Outrage spreads faster than reasonableness on social media, so a story about \\nhow an employer underpaid someone gets amplified, whereas stories about how an employer \\ntreated someone fairly do not. The vast majority of employers are ethical and fair, so don’t let \\nstories about the small fraction of mistreated individuals sway your approach. If you’re leaving \\na job, exit gracefully. Give your employer ample notice, give your full effort through your last \\nhour on the job, transition unfinished business as best you can, and leave in a way that honors \\nthe responsibilities you were entrusted with.Finding the Right AI Job For You CHAPTER 9\\n\\nPAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll work \\non. But the teammates you’ll work with are at least equally important. We’re influenced by \\npeople around us, so your colleagues will make a big difference. For example, if your friends \\nsmoke, the odds increase that you, too, will smoke. I don’t know of a study that shows this, \\nbut I’m pretty sure that if most of your colleagues work hard, learn continuously, and build AI \\nto benefit all people, you’re likely to do the same. (By the way, some large companies won’t \\ntell you who your teammates will be until you’ve accepted an offer. In this case, be persistent \\nand keep pushing to identify and speak with potential teammates. Strict policies may make it \\nimpossible to accommodate you, but in my mind, that increases the risk of accepting the offer, \\nas it increases the odds you’ll end up with a manager or teammates who aren’t a good fit.)\\nGet help from your community. Most of us go job hunting only a small number of times in our \\ncareers, so few of us get much practice at doing it well. Collectively, though, people in your \\nimmediate community probably have a lot of experience. Don’t be shy about calling on them. \\nFriends and associates can provide advice, share inside knowledge, and refer you to others \\nwho may help. I got a lot of help from supportive friends and mentors when I applied for my \\nfirst faculty position, and many of the tips they gave me were very helpful.\\nI know that the job-search process can be intimidating. Instead of viewing it as a great leap, \\nconsider an incremental approach. Start by identifying possible roles and conducting a handful \\nof informational interviews. If these conversations tell you that you have more learning to do \\nbefore you’re ready to apply, that’s great! At least you have a clear path forward. The most \\nimportant part of any journey is to take the first step, and that step can be a small one.Finding the Right AI Job For You CHAPTER 9\\n\\nPAGE 34Keys to Building a \\nCareer in AICHAPTER 10\\nJOBS\\n\\nPAGE 35Keys to Building a Career in AI CHAPTER 10\\nThe path to career success in AI is more complex than what I can  cover in one short eBook. \\nHopefully the previous chapters will give you momentum to move forward. \\nHere are additional things to think about as you plot your path to success: \\nWhen we tackle large projects, we succeed better by \\nworking in teams than individually. The ability to collaborate \\nwith, influence, and be influenced by others is critical. \\nThus, interpersonal and communication skills really matter. \\n(I used to be a pretty bad communicator, by the way.)1. Teamwork:\\nI hate networking! As an introvert, having to go to a party \\nto smile and shake as many hands as possible is an activity \\nthat borders on horrific. I’d much rather stay home and read \\na book. Nonetheless, I’m fortunate to have found many \\ngenuine friends in AI; people I would gladly go to bat for \\nand who I count on as well. No person is an island, and \\nhaving a strong professional network can help propel you \\nforward in the moments when you need help or advice. In \\nlieu of networking, I’ve found it more helpful to think about \\nbuilding up a community. So instead of trying to build up \\nmy personal network, I focus instead on building up the \\ncommunities that I’m part of. This has the side effect of \\nhelping me meet more people and make friends as well. 2. Networking:\\n\\n\\nPAGE 36Keys to Building a Career in AI CHAPTER 10\\nOf all the steps in building a career, this \\none tends to receive the most attention. \\nUnfortunately, there is a lot of bad advice \\nabout this on the internet. (For example, many \\narticles urge taking an adversarial attitude \\ntoward potential employers, which I don’t think \\nis helpful.) Although it may seem like finding a \\njob is the ultimate goal, it’s just one small step \\nin the long journey of a career.3. Job search\\nFew people will know whether you spend \\nyour weekends learning, or binge watching \\nTV — but they will notice the difference over \\ntime. Many successful people develop good \\nhabits in eating, exercise, sleep, personal \\nrelationships, work, learning, and self-care. \\nSuch habits help them move forward while \\nstaying healthy.4. Personal discipline\\nI find that people  who aim to lift others during \\nevery step of their own journey often achieve \\nbetter outcomes for themselves. How can we \\nhelp others even as we build an exciting career \\nfor ourselves?5. Altruism\\n\\n\\nPAGE 37Overcoming Imposter \\nSyndromeCHAPTER 11\\n\\nPAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter of \\nnewcomers to AI sometimes experiencing imposter syndrome, where someone — regardless \\nof their success in the field — wonders if they’re a fraud and really belong in the AI community. \\nI want to make sure this doesn’t discourage you or anyone else from growing in AI.\\nAn estimated 70 percent of people experience some form of imposter syndrome at some point. \\nMany talented people have spoken publicly about this experience, including former Facebook \\nCOO Sheryl Sandberg, U.S. first lady Michelle Obama, actor Tom Hanks, and Atlassian co-CEO \\nMike Cannon-Brookes. It happens in our community even among accomplished people. If you’ve \\nnever experienced this yourself, that’s great! I hope you’ll join me in encouraging and welcoming \\neveryone who wants to join our community.\\nAI is technically complex, and it has its fair share of smart and highly capable people. But it is \\neasy to forget that to become good at anything, the first step is to suck at it. If you’ve succeeded \\nat sucking at AI — congratulations, you’re on your way!\\nI once struggled to understand the math behind linear regression. I was mystified when \\nlogistic regression performed strangely on my data, and it took me days to find a bug in my \\nimplementation of a basic neural network. Today, I still find many research papers challenging \\nto read, and I recently made an obvious mistake while tuning a neural network hyperparameter \\n(that fortunately a fellow engineer caught and fixed).\\nSo if you, too, find parts of AI challenging, it’s okay. We’ve all been there. I guarantee that everyone \\nwho has published a seminal AI paper struggled with similar technical challenges at some point.Let me be clear: If you want to be part of the AI \\ncommunity, then I welcome you with open arms. \\nIf you want to join us, you fully belong with us!Overcoming Imposter Syndrome\\n\\nPAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to my \\none-year-old son. No matter how far along you are — if you’re at least as knowledgeable as a \\nthree-year-old — you can encourage and lift up others behind you. Doing so will help you, too, \\nas others behind you will recognize your expertise and also encourage you to keep developing. \\nWhen you invite others to join the AI community, which I hope you will do, it also reduces any \\ndoubts that you are already one of us.\\nAI is such an important part of our world that I would like everyone who wants to be part of it \\nto feel at home as a member of our community. Let’s work together to make it happen.Here are some things that can help.\\nDo you have supportive mentors or peers? If you don’t yet, attend Pie & AI or other events, \\nuse discussion boards, and work on finding some. If your mentors or manager don’t support \\nyour growth, find ones who do. I’m also working on how to grow a supportive AI community \\nand hope to make finding and giving support easier for everyone.\\nNo one is an expert at everything. Recognize what you do well. If what you do well is \\nunderstand and explain to your friends one-tenth of the articles in The Batch,  then you’re \\non your way! Let’s work on getting you to understand two-tenths of the articles.✓\\n✓\\nOvercoming Imposter Syndrome\\n\\nPAGE 40Make Every Day CountFinal Thoughts\\nEvery year on my birthday, I get to thinking about the days behind and those \\nthat may lie ahead.\\nWhen I ask friends, many choose a number in the hundreds of \\nthousands. (Many others can’t resist calculating the answer, \\nto my annoyance!)\\nWhen I was a grad student, I remember plugging my statistics \\ninto a mortality calculator to figure out my life expectancy. \\nThe calculator said I could expect to live a total of 27,649 \\ndays. It struck me how small this number is. I printed it in a \\nlarge font and pasted it on my office wall as a daily reminder.\\nThat’s all the days we have to spend with loved ones, learn, \\nbuild for the future, and help others. Whatever you’re doing \\ntoday, is it worth 1/30,000 of your life?How many days is a \\ntypical human lifespan?Maybe you’re good at math; I’m sure you’ll be able to answer the following question \\nvia a quick calculation. But let me ask you a question, and please answer from \\nyour gut, without calculating.\\n20,000 days 100,000 days\\n1 million days 5 million days\\n\\nPAGE 41\\n', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from llama_index import Document\n",
    "\n",
    "document = Document(text=\"\\n\\n\".\\\n",
    "                    join([doc.text for doc in documents]))\n",
    "document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "height": 217
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /tmp/llama_index...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a8cc425bcbd449ac856067cfe0b801d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/743 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6cf869b55c7d45019926169b0d389004",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/133M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94d1e64e6a9742788570013643159afc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/366 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c338ca918a4b48268509604bfa76d52d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8696e8123f2a46b5ba3de1a3a157ab68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/711k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "707531ce8cfc42edb060eb4bff4393b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/125 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from utils import build_sentence_window_index\n",
    "\n",
    "from llama_index.llms import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)\n",
    "\n",
    "sentence_index = build_sentence_window_index(\n",
    "    document,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    save_dir=\"sentence_index\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "height": 81
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b07d50e7d7304c28ad8e2b5246be460d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/799 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2170b7817a5442438b15487f2c6a9763",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/1.11G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8323469bfd274899b06bbad3259fa207",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/443 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c7a79c07168740a2ae2926d8efea7052",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "sentencepiece.bpe.model:   0%|          | 0.00/5.07M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d200fdf0e1f141208ff92618a872d753",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/17.1M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f56f2e090a64b62a7bd7871529a070a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/279 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<llama_index.query_engine.retriever_query_engine.RetrieverQueryEngine at 0x7f5451063280>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from utils import get_sentence_window_query_engine\n",
    "\n",
    "sentence_window_engine = \\\n",
    "get_sentence_window_query_engine(sentence_index)\n",
    "sentence_window_engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Response(response='You create your AI portfolio by showcasing a progression of projects that demonstrate your skills.', source_nodes=[NodeWithScore(node=TextNode(id_='19cac35e-ea40-4f81-83f8-c52bd258317d', embedding=None, metadata={'window': 'Chapter 4: Scoping Successful AI Projects.\\n Chapter 5: Finding Projects that Complement \\nYour Career Goals.\\n Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n', 'original_text': 'Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n'}, excluded_embed_metadata_keys=['window', 'original_text'], excluded_llm_metadata_keys=['window', 'original_text'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='3750e7b2-dd97-4d8f-a87e-bbd70b908260', node_type=<ObjectType.DOCUMENT: '4'>, metadata={}, hash='fdc7ce9a017ce62d4e0b30cc419051a9bb56618a8569f51a2b64f8535fe6244e'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='fd0b1202-6bd2-40b1-942c-1912e0771c41', node_type=<ObjectType.TEXT: '1'>, metadata={'window': 'Chapter 3: Should You Learn Math to Get a Job \\nin AI?\\n Chapter 4: Scoping Successful AI Projects.\\n Chapter 5: Finding Projects that Complement \\nYour Career Goals.\\n Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n', 'original_text': 'Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n'}, hash='0b2ab77d43433bf32079de5550147d7803fbd43511bc54e455621e83ff9831fd'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='c64908c5-34a0-4a2b-9bb3-2200eb972ed4', node_type=<ObjectType.TEXT: '1'>, metadata={'window': 'Chapter 5: Finding Projects that Complement \\nYour Career Goals.\\n Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n Chapter 10: Keys to Building a Career in AI.\\n', 'original_text': 'Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n'}, hash='088552c5fc3bc7c67d373579dc176b18597f51e87304a70de81c191a128d99f1')}, hash='2b389c732f348913dc3772abd5d24c75c779563240881173dd420b2a84535193', text='Chapter 4: Scoping Successful AI Projects.\\n Chapter 5: Finding Projects that Complement \\nYour Career Goals.\\n Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n', start_char_idx=627, end_char_idx=691, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'), score=0.56953126), NodeWithScore(node=TextNode(id_='a3f506b3-d777-48fd-a0b8-e122260dee11', embedding=None, metadata={'window': 'Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n Chapter 10: Keys to Building a Career in AI.\\n Chapter 11: Overcoming Imposter Syndrome.\\n', 'original_text': 'Chapter 9: Finding the Right AI Job for You.\\n'}, excluded_embed_metadata_keys=['window', 'original_text'], excluded_llm_metadata_keys=['window', 'original_text'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='3750e7b2-dd97-4d8f-a87e-bbd70b908260', node_type=<ObjectType.DOCUMENT: '4'>, metadata={}, hash='fdc7ce9a017ce62d4e0b30cc419051a9bb56618a8569f51a2b64f8535fe6244e'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='c64908c5-34a0-4a2b-9bb3-2200eb972ed4', node_type=<ObjectType.TEXT: '1'>, metadata={'window': 'Chapter 5: Finding Projects that Complement \\nYour Career Goals.\\n Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n Chapter 10: Keys to Building a Career in AI.\\n', 'original_text': 'Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n'}, hash='088552c5fc3bc7c67d373579dc176b18597f51e87304a70de81c191a128d99f1'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='b238eb88-e6bd-45c7-8299-bc1402dae0e8', node_type=<ObjectType.TEXT: '1'>, metadata={'window': 'Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n Chapter 10: Keys to Building a Career in AI.\\n Chapter 11: Overcoming Imposter Syndrome.\\n Final Thoughts: Make Every Day Count.LEARNING\\nPROJECTS\\nJOB\\n\\nPAGE 4Coding AI Is the New Literacy\\nToday we take it for granted that many people know how to read and write. ', 'original_text': 'Chapter 10: Keys to Building a Career in AI.\\n'}, hash='3f2164a7c078543209081746cb6d27baffac0a7325f4cda09a454d95b0c06d8f')}, hash='09819dfc69470ce117a92fd4b0140d07552e9eec0745e9837e0857a9e52c53be', text='Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n Chapter 10: Keys to Building a Career in AI.\\n Chapter 11: Overcoming Imposter Syndrome.\\n', start_char_idx=757, end_char_idx=802, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'), score=0.5678981)], metadata={'19cac35e-ea40-4f81-83f8-c52bd258317d': {'window': 'Chapter 4: Scoping Successful AI Projects.\\n Chapter 5: Finding Projects that Complement \\nYour Career Goals.\\n Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n', 'original_text': 'Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n'}, 'a3f506b3-d777-48fd-a0b8-e122260dee11': {'window': 'Chapter 6: Building a Portfolio of Projects that \\nShows Skill Progression.\\n Chapter 7: A Simple Framework for Starting Your AI \\nJob Search.\\n Chapter 8: Using Informational Interviews to Find \\nthe Right Job.\\n Chapter 9: Finding the Right AI Job for You.\\n Chapter 10: Keys to Building a Career in AI.\\n Chapter 11: Overcoming Imposter Syndrome.\\n', 'original_text': 'Chapter 9: Finding the Right AI Job for You.\\n'}})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = sentence_window_engine.query(\n",
    "    \"How do you create your AI portfolio?\")\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You create your AI portfolio by showcasing a progression of projects that demonstrate your skills.\n"
     ]
    }
   ],
   "source": [
    "print(output.response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedback functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "height": 64,
    "id": "5KqV-IbQaZVB"
   },
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "from trulens_eval import OpenAI as fOpenAI\n",
    "\n",
    "provider = fOpenAI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OpenAI(endpoint=OpenAIEndpoint(name='openai', rpm=60, retries=3, post_headers={}, pace=<queue.Queue object at 0x7f545110fc10>, global_callback=OpenAICallback(cost=Cost(n_requests=0, n_successful_requests=0, n_classes=0, n_tokens=0, n_stream_chunks=0, n_prompt_tokens=0, n_completion_tokens=0, cost=0.0), langchain_handler=Tokens Used: 0\n",
       "\tPrompt Tokens: 0\n",
       "\tCompletion Tokens: 0\n",
       "Successful Requests: 0\n",
       "Total Cost (USD): $0.0, chunks=[]), callback_class=<class 'trulens_eval.feedback.provider.endpoint.openai.OpenAICallback'>, callback_name='callback_openai', pace_thread=<Thread(Thread-7 (keep_pace), started daemon 140000111625920)>, client=<openai.OpenAI object at 0x7f54510632b0>), model_engine='gpt-3.5-turbo')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "provider"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Answer Relevance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "height": 115
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Feedback(implementation=Method(obj=ObjSerial(cls=trulens_eval.feedback.provider.openai.OpenAI, id=140000114203840, init_bindings=Bindings(args=(), kwargs={'endpoint': {'name': 'openai', 'rpm': 60, 'retries': 3, 'client': <openai.OpenAI object at 0x7f54510632b0>}, 'model_engine': 'gpt-3.5-turbo'})), name='relevance_with_cot_reasons'), aggregator=Function(module=Module(package_name='numpy', module_name='numpy'), cls=None, name='mean'), feedback_definition_id='feedback_definition_hash_4f65a0cb3e4e1e17fed778ce357d1245', selectors={'prompt': Lens().__record__.main_input, 'response': Lens().__record__.main_output}, supplied_name='Answer Relevance', imp=<bound method LLMProvider.relevance_with_cot_reasons of OpenAI(endpoint=OpenAIEndpoint(name='openai', rpm=60, retries=3, post_headers={}, pace=<queue.Queue object at 0x7f545110fc10>, global_callback=OpenAICallback(cost=Cost(n_requests=0, n_successful_requests=0, n_classes=0, n_tokens=0, n_stream_chunks=0, n_prompt_tokens=0, n_completion_tokens=0, cost=0.0), langchain_handler=Tokens Used: 0\n",
       "\tPrompt Tokens: 0\n",
       "\tCompletion Tokens: 0\n",
       "Successful Requests: 0\n",
       "Total Cost (USD): $0.0, chunks=[]), callback_class=<class 'trulens_eval.feedback.provider.endpoint.openai.OpenAICallback'>, callback_name='callback_openai', pace_thread=<Thread(Thread-7 (keep_pace), started daemon 140000111625920)>, client=<openai.OpenAI object at 0x7f54510632b0>), model_engine='gpt-3.5-turbo')>, agg=<function mean at 0x7f54e826d6c0>, higher_is_better=True)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from trulens_eval import Feedback\n",
    "\n",
    "f_qa_relevance = Feedback(\n",
    "    provider.relevance_with_cot_reasons,\n",
    "    name=\"Answer Relevance\"\n",
    ").on_input_output()\n",
    "f_qa_relevance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Context Relevance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Lens().__record__.app.query.rets.source_nodes[:].node.text"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from trulens_eval import TruLlama\n",
    "\n",
    "context_selection = TruLlama.select_source_nodes().node.text\n",
    "context_selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "height": 166
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In Context Relevance, input question will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Context Relevance, input statement will be set to __record__.app.query.rets.source_nodes[:].node.text .\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "f_qs_relevance = (\n",
    "    Feedback(provider.qs_relevance,\n",
    "             name=\"Context Relevance\")\n",
    "    .on_input()\n",
    "    .on(context_selection)\n",
    "    .aggregate(np.mean)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "height": 166
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In Context Relevance, input question will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Context Relevance, input statement will be set to __record__.app.query.rets.source_nodes[:].node.text .\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "f_qs_relevance = (\n",
    "    Feedback(provider.qs_relevance_with_cot_reasons,\n",
    "             name=\"Context Relevance\")\n",
    "    .on_input()\n",
    "    .on(context_selection)\n",
    "    .aggregate(np.mean)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Groundedness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "from trulens_eval.feedback import Groundedness\n",
    "\n",
    "grounded = Groundedness(groundedness_provider=provider)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "height": 149,
    "id": "kXJBD4gfaZVC"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text .\n",
      "✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .\n"
     ]
    }
   ],
   "source": [
    "f_groundedness = (\n",
    "    Feedback(grounded.groundedness_measure_with_cot_reasons,\n",
    "             name=\"Groundedness\"\n",
    "            )\n",
    "    .on(context_selection)\n",
    "    .on_output()\n",
    "    .aggregate(grounded.grounded_statements_aggregator)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation of the RAG application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "height": 217,
    "id": "KUDHInR-aZVC"
   },
   "outputs": [],
   "source": [
    "from trulens_eval import TruLlama\n",
    "from trulens_eval import FeedbackMode\n",
    "\n",
    "tru_recorder = TruLlama(\n",
    "    sentence_window_engine,\n",
    "    app_id=\"App_1\",\n",
    "    feedbacks=[\n",
    "        f_qa_relevance,\n",
    "        f_qs_relevance,\n",
    "        f_groundedness\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "height": 115,
    "id": "dsA3ziw1aZVD"
   },
   "outputs": [],
   "source": [
    "eval_questions = []\n",
    "with open('eval_questions.txt', 'r') as file:\n",
    "    for line in file:\n",
    "        # Remove newline character and convert to integer\n",
    "        item = line.strip()\n",
    "        eval_questions.append(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['What are the keys to building a career in AI?',\n",
       " 'How can teamwork contribute to success in AI?',\n",
       " 'What is the importance of networking in AI?',\n",
       " 'What are some good habits to develop for a successful career?',\n",
       " 'How can altruism be beneficial in building a career?',\n",
       " 'What is imposter syndrome and how does it relate to AI?',\n",
       " 'Who are some accomplished individuals who have experienced imposter syndrome?',\n",
       " 'What is the first step to becoming good at AI?',\n",
       " 'What are some common challenges in AI?',\n",
       " 'Is it normal to find parts of AI challenging?']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "eval_questions.append(\"How can I be successful in AI?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['What are the keys to building a career in AI?',\n",
       " 'How can teamwork contribute to success in AI?',\n",
       " 'What is the importance of networking in AI?',\n",
       " 'What are some good habits to develop for a successful career?',\n",
       " 'How can altruism be beneficial in building a career?',\n",
       " 'What is imposter syndrome and how does it relate to AI?',\n",
       " 'Who are some accomplished individuals who have experienced imposter syndrome?',\n",
       " 'What is the first step to becoming good at AI?',\n",
       " 'What are some common challenges in AI?',\n",
       " 'Is it normal to find parts of AI challenging?',\n",
       " 'How can I be successful in AI?']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "height": 64,
    "id": "01_P6TxaaZVD"
   },
   "outputs": [],
   "source": [
    "for question in eval_questions:\n",
    "    with tru_recorder as recording:\n",
    "        sentence_window_engine.query(question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "height": 47,
    "id": "sNPhDde6ZArq"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_id</th>\n",
       "      <th>app_json</th>\n",
       "      <th>type</th>\n",
       "      <th>record_id</th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "      <th>tags</th>\n",
       "      <th>record_json</th>\n",
       "      <th>cost_json</th>\n",
       "      <th>perf_json</th>\n",
       "      <th>ts</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance_calls</th>\n",
       "      <th>Context Relevance_calls</th>\n",
       "      <th>Groundedness_calls</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_tokens</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>App_1</td>\n",
       "      <td>{\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_a346bdd57988d75b1ed5748c99f9506f</td>\n",
       "      <td>\"What are the keys to building a career in AI?\"</td>\n",
       "      <td>\"The keys to building a career in AI involve l...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_a346bdd57988d75b1ed...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-11-23T07:44:11.054218\", \"...</td>\n",
       "      <td>2024-11-23T07:44:12.570911</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>[{'args': {'prompt': 'What are the keys to bui...</td>\n",
       "      <td>[{'args': {'question': 'What are the keys to b...</td>\n",
       "      <td>[{'args': {'source': 'Chapter 7: A Simple Fram...</td>\n",
       "      <td>1</td>\n",
       "      <td>509</td>\n",
       "      <td>0.000779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>App_1</td>\n",
       "      <td>{\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_0a5ee7d8bcd85b16bd00a9ad515d3679</td>\n",
       "      <td>\"How can teamwork contribute to success in AI?\"</td>\n",
       "      <td>\"Teammates play a crucial role in the success ...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_0a5ee7d8bcd85b16bd0...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-11-23T07:44:12.702083\", \"...</td>\n",
       "      <td>2024-11-23T07:44:14.789513</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>[{'args': {'prompt': 'How can teamwork contrib...</td>\n",
       "      <td>[{'args': {'question': 'How can teamwork contr...</td>\n",
       "      <td>[{'args': {'source': 'To get \n",
       "a project starte...</td>\n",
       "      <td>2</td>\n",
       "      <td>635</td>\n",
       "      <td>0.000992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>App_1</td>\n",
       "      <td>{\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_f186a84ffdfc340006b60b8e00b2afd5</td>\n",
       "      <td>\"What is the importance of networking in AI?\"</td>\n",
       "      <td>\"Networking in AI is crucial as it can provide...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_f186a84ffdfc340006b...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-11-23T07:44:14.918873\", \"...</td>\n",
       "      <td>2024-11-23T07:44:17.176066</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>[{'args': {'prompt': 'What is the importance o...</td>\n",
       "      <td>[{'args': {'question': 'What is the importance...</td>\n",
       "      <td>[{'args': {'source': 'What is the hiring proce...</td>\n",
       "      <td>2</td>\n",
       "      <td>538</td>\n",
       "      <td>0.000859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>App_1</td>\n",
       "      <td>{\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_e759028ea1dcd896b5a3f853af56e444</td>\n",
       "      <td>\"What are some good habits to develop for a su...</td>\n",
       "      <td>\"Developing good habits in areas such as eatin...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_e759028ea1dcd896b5a...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-11-23T07:44:17.295818\", \"...</td>\n",
       "      <td>2024-11-23T07:44:19.117637</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>[{'args': {'prompt': 'What are some good habit...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'args': {'source': '(For example, many \n",
       "arti...</td>\n",
       "      <td>1</td>\n",
       "      <td>454</td>\n",
       "      <td>0.000710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>App_1</td>\n",
       "      <td>{\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_e51ab7184bf9ea82fa019acaf436afd8</td>\n",
       "      <td>\"How can altruism be beneficial in building a ...</td>\n",
       "      <td>\"Helping others and lifting them up during one...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_e51ab7184bf9ea82fa0...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-11-23T07:44:19.233066\", \"...</td>\n",
       "      <td>2024-11-23T07:44:20.550482</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'args': {'prompt': 'How can altruism be bene...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>482</td>\n",
       "      <td>0.000734</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  app_id                                           app_json  \\\n",
       "0  App_1  {\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...   \n",
       "1  App_1  {\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...   \n",
       "2  App_1  {\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...   \n",
       "3  App_1  {\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...   \n",
       "4  App_1  {\"app_id\": \"App_1\", \"tags\": \"-\", \"metadata\": {...   \n",
       "\n",
       "                                                type  \\\n",
       "0  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "1  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "2  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "3  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "4  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "\n",
       "                                      record_id  \\\n",
       "0  record_hash_a346bdd57988d75b1ed5748c99f9506f   \n",
       "1  record_hash_0a5ee7d8bcd85b16bd00a9ad515d3679   \n",
       "2  record_hash_f186a84ffdfc340006b60b8e00b2afd5   \n",
       "3  record_hash_e759028ea1dcd896b5a3f853af56e444   \n",
       "4  record_hash_e51ab7184bf9ea82fa019acaf436afd8   \n",
       "\n",
       "                                               input  \\\n",
       "0    \"What are the keys to building a career in AI?\"   \n",
       "1    \"How can teamwork contribute to success in AI?\"   \n",
       "2      \"What is the importance of networking in AI?\"   \n",
       "3  \"What are some good habits to develop for a su...   \n",
       "4  \"How can altruism be beneficial in building a ...   \n",
       "\n",
       "                                              output tags  \\\n",
       "0  \"The keys to building a career in AI involve l...    -   \n",
       "1  \"Teammates play a crucial role in the success ...    -   \n",
       "2  \"Networking in AI is crucial as it can provide...    -   \n",
       "3  \"Developing good habits in areas such as eatin...    -   \n",
       "4  \"Helping others and lifting them up during one...    -   \n",
       "\n",
       "                                         record_json  \\\n",
       "0  {\"record_id\": \"record_hash_a346bdd57988d75b1ed...   \n",
       "1  {\"record_id\": \"record_hash_0a5ee7d8bcd85b16bd0...   \n",
       "2  {\"record_id\": \"record_hash_f186a84ffdfc340006b...   \n",
       "3  {\"record_id\": \"record_hash_e759028ea1dcd896b5a...   \n",
       "4  {\"record_id\": \"record_hash_e51ab7184bf9ea82fa0...   \n",
       "\n",
       "                                           cost_json  \\\n",
       "0  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "1  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "2  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "3  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "4  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "\n",
       "                                           perf_json  \\\n",
       "0  {\"start_time\": \"2024-11-23T07:44:11.054218\", \"...   \n",
       "1  {\"start_time\": \"2024-11-23T07:44:12.702083\", \"...   \n",
       "2  {\"start_time\": \"2024-11-23T07:44:14.918873\", \"...   \n",
       "3  {\"start_time\": \"2024-11-23T07:44:17.295818\", \"...   \n",
       "4  {\"start_time\": \"2024-11-23T07:44:19.233066\", \"...   \n",
       "\n",
       "                           ts  Answer Relevance  Context Relevance  \\\n",
       "0  2024-11-23T07:44:12.570911               1.0                0.5   \n",
       "1  2024-11-23T07:44:14.789513               0.9                0.8   \n",
       "2  2024-11-23T07:44:17.176066               1.0                0.5   \n",
       "3  2024-11-23T07:44:19.117637               1.0                NaN   \n",
       "4  2024-11-23T07:44:20.550482               1.0                NaN   \n",
       "\n",
       "   Groundedness                             Answer Relevance_calls  \\\n",
       "0      0.500000  [{'args': {'prompt': 'What are the keys to bui...   \n",
       "1      0.666667  [{'args': {'prompt': 'How can teamwork contrib...   \n",
       "2      0.300000  [{'args': {'prompt': 'What is the importance o...   \n",
       "3      1.000000  [{'args': {'prompt': 'What are some good habit...   \n",
       "4           NaN  [{'args': {'prompt': 'How can altruism be bene...   \n",
       "\n",
       "                             Context Relevance_calls  \\\n",
       "0  [{'args': {'question': 'What are the keys to b...   \n",
       "1  [{'args': {'question': 'How can teamwork contr...   \n",
       "2  [{'args': {'question': 'What is the importance...   \n",
       "3                                                NaN   \n",
       "4                                                NaN   \n",
       "\n",
       "                                  Groundedness_calls  latency  total_tokens  \\\n",
       "0  [{'args': {'source': 'Chapter 7: A Simple Fram...        1           509   \n",
       "1  [{'args': {'source': 'To get \n",
       "a project starte...        2           635   \n",
       "2  [{'args': {'source': 'What is the hiring proce...        2           538   \n",
       "3  [{'args': {'source': '(For example, many \n",
       "arti...        1           454   \n",
       "4                                                NaN        1           482   \n",
       "\n",
       "   total_cost  \n",
       "0    0.000779  \n",
       "1    0.000992  \n",
       "2    0.000859  \n",
       "3    0.000710  \n",
       "4    0.000734  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "records, feedback = tru.get_records_and_feedback(app_ids=[])\n",
    "records.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Context Relevance', 'Groundedness', 'Answer Relevance']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feedback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "height": 81
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"What are the keys to building a career in AI?\"</td>\n",
       "      <td>\"The keys to building a career in AI involve learning foundational technical skills, working on projects, finding a job, and being part of a supportive community.\"</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"How can teamwork contribute to success in AI?\"</td>\n",
       "      <td>\"Teammates play a crucial role in the success of AI projects. Working collaboratively with colleagues who are dedicated, continuously learning, and focused on using AI to benefit society can positively influence one's own work ethic and outcomes. The ability to work effectively in a team, leveraging each member's strengths and insights, can lead to improved project outcomes and personal growth as a leader in AI projects.\"</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"What is the importance of networking in AI?\"</td>\n",
       "      <td>\"Networking in AI is crucial as it can provide valuable insights, guidance, and opportunities for individuals looking to advance in the field. By connecting with professionals who have experience in AI, individuals can gain knowledge about the industry, potential career paths, and current trends. Networking also allows individuals to build relationships with others in the field, which can lead to mentorship, collaboration on projects, and even job opportunities. Additionally, networking can help individuals stay informed about the latest developments in AI and expand their professional connections for future career growth.\"</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"What are some good habits to develop for a successful career?\"</td>\n",
       "      <td>\"Developing good habits in areas such as eating, exercise, sleep, personal relationships, work, learning, and self-care can help individuals move forward in their careers while maintaining their health. Additionally, aiming to lift others during every step of one's own journey can lead to better outcomes.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"How can altruism be beneficial in building a career?\"</td>\n",
       "      <td>\"Helping others and lifting them up during one's own career journey can lead to better outcomes for oneself.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\"What is imposter syndrome and how does it relate to AI?\"</td>\n",
       "      <td>\"Imposter syndrome is a phenomenon where individuals doubt their accomplishments and have a persistent fear of being exposed as a fraud, despite evidence of their success. In the context of AI, newcomers to the field may experience imposter syndrome, feeling like they do not truly belong in the AI community even if they have achieved success. This can be a common experience for individuals in AI, as highlighted by the mention of imposter syndrome among newcomers to the field.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\"Who are some accomplished individuals who have experienced imposter syndrome?\"</td>\n",
       "      <td>\"Former Facebook COO Sheryl Sandberg, U.S. first lady Michelle Obama, actor Tom Hanks, and Atlassian co-CEO Mike Cannon-Brookes are some accomplished individuals who have experienced imposter syndrome.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\"What is the first step to becoming good at AI?\"</td>\n",
       "      <td>\"The first step to becoming good at AI is learning foundational technical skills.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>\"What are some common challenges in AI?\"</td>\n",
       "      <td>\"Some common challenges in AI include keeping up-to-date with rapidly evolving technologies, the iterative nature of AI projects leading to uncertainties in project management, the need to collaborate with stakeholders who may lack expertise in AI, difficulties in finding suitable projects and estimating timelines, and the challenge of tuning neural network hyperparameters accurately.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>\"Is it normal to find parts of AI challenging?\"</td>\n",
       "      <td>\"It is normal to find parts of AI challenging.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>\"How can I be successful in AI?\"</td>\n",
       "      <td>\"To be successful in AI, it is important to understand the problem thoroughly in order to brainstorm potential solutions efficiently. Determining milestones and metrics for the project is crucial once its value has been established. Additionally, having a basic intuitive understanding of calculus can be beneficial, as it plays a role in machine learning. Continuous learning and deepening technical knowledge in specific application or technology areas within AI can also contribute to success in the field.\"</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                              input  \\\n",
       "0                                   \"What are the keys to building a career in AI?\"   \n",
       "1                                   \"How can teamwork contribute to success in AI?\"   \n",
       "2                                     \"What is the importance of networking in AI?\"   \n",
       "3                   \"What are some good habits to develop for a successful career?\"   \n",
       "4                            \"How can altruism be beneficial in building a career?\"   \n",
       "5                         \"What is imposter syndrome and how does it relate to AI?\"   \n",
       "6   \"Who are some accomplished individuals who have experienced imposter syndrome?\"   \n",
       "7                                  \"What is the first step to becoming good at AI?\"   \n",
       "8                                          \"What are some common challenges in AI?\"   \n",
       "9                                   \"Is it normal to find parts of AI challenging?\"   \n",
       "10                                                 \"How can I be successful in AI?\"   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     output  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       \"The keys to building a career in AI involve learning foundational technical skills, working on projects, finding a job, and being part of a supportive community.\"   \n",
       "1                                                                                                                                                                                                                 \"Teammates play a crucial role in the success of AI projects. Working collaboratively with colleagues who are dedicated, continuously learning, and focused on using AI to benefit society can positively influence one's own work ethic and outcomes. The ability to work effectively in a team, leveraging each member's strengths and insights, can lead to improved project outcomes and personal growth as a leader in AI projects.\"   \n",
       "2   \"Networking in AI is crucial as it can provide valuable insights, guidance, and opportunities for individuals looking to advance in the field. By connecting with professionals who have experience in AI, individuals can gain knowledge about the industry, potential career paths, and current trends. Networking also allows individuals to build relationships with others in the field, which can lead to mentorship, collaboration on projects, and even job opportunities. Additionally, networking can help individuals stay informed about the latest developments in AI and expand their professional connections for future career growth.\"   \n",
       "3                                                                                                                                                                                                                                                                                                                                       \"Developing good habits in areas such as eating, exercise, sleep, personal relationships, work, learning, and self-care can help individuals move forward in their careers while maintaining their health. Additionally, aiming to lift others during every step of one's own journey can lead to better outcomes.\"   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \"Helping others and lifting them up during one's own career journey can lead to better outcomes for oneself.\"   \n",
       "5                                                                                                                                                         \"Imposter syndrome is a phenomenon where individuals doubt their accomplishments and have a persistent fear of being exposed as a fraud, despite evidence of their success. In the context of AI, newcomers to the field may experience imposter syndrome, feeling like they do not truly belong in the AI community even if they have achieved success. This can be a common experience for individuals in AI, as highlighted by the mention of imposter syndrome among newcomers to the field.\"   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                \"Former Facebook COO Sheryl Sandberg, U.S. first lady Michelle Obama, actor Tom Hanks, and Atlassian co-CEO Mike Cannon-Brookes are some accomplished individuals who have experienced imposter syndrome.\"   \n",
       "7                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \"The first step to becoming good at AI is learning foundational technical skills.\"   \n",
       "8                                                                                                                                                                                                                                                      \"Some common challenges in AI include keeping up-to-date with rapidly evolving technologies, the iterative nature of AI projects leading to uncertainties in project management, the need to collaborate with stakeholders who may lack expertise in AI, difficulties in finding suitable projects and estimating timelines, and the challenge of tuning neural network hyperparameters accurately.\"   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \"It is normal to find parts of AI challenging.\"   \n",
       "10                                                                                                                           \"To be successful in AI, it is important to understand the problem thoroughly in order to brainstorm potential solutions efficiently. Determining milestones and metrics for the project is crucial once its value has been established. Additionally, having a basic intuitive understanding of calculus can be beneficial, as it plays a role in machine learning. Continuous learning and deepening technical knowledge in specific application or technology areas within AI can also contribute to success in the field.\"   \n",
       "\n",
       "    Context Relevance  Groundedness  Answer Relevance  \n",
       "0                 0.5      0.500000               1.0  \n",
       "1                 0.8      0.666667               0.9  \n",
       "2                 0.5      0.300000               1.0  \n",
       "3                 NaN      1.000000               1.0  \n",
       "4                 NaN           NaN               1.0  \n",
       "5                 NaN           NaN               1.0  \n",
       "6                 NaN           NaN               1.0  \n",
       "7                 NaN           NaN               1.0  \n",
       "8                 NaN           NaN               NaN  \n",
       "9                 NaN           NaN               NaN  \n",
       "10                NaN           NaN               NaN  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "records[[\"input\", \"output\"] + feedback]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>app_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>App_1</th>\n",
       "      <td>0.581818</td>\n",
       "      <td>0.761364</td>\n",
       "      <td>0.990909</td>\n",
       "      <td>1.454545</td>\n",
       "      <td>0.000847</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Context Relevance  Groundedness  Answer Relevance   latency  \\\n",
       "app_id                                                                \n",
       "App_1            0.581818      0.761364          0.990909  1.454545   \n",
       "\n",
       "        total_cost  \n",
       "app_id              \n",
       "App_1     0.000847  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tru.get_leaderboard(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 30,
    "id": "6Yp4_e4faZVD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting dashboard ...\n",
      "Config file already exists. Skipping writing process.\n",
      "Credentials file already exists. Skipping writing process.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc72aff75edc47abbe0a71828ec4e2bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(VBox(children=(Label(value='STDOUT'), Output())), VBox(children=(Label(valu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tru.run_dashboard()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": [
    {
     "file_id": "1ChKW1kEIUcUVDDTWjpA5Tf_ib3Hhp3uS",
     "timestamp": 1695164681916
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
